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ABSTRACT 


This document contains an introduction to the interprocess communication 

facilities included in the 4.2bsd release of the VAX* UNIX** system, as distri¬ 
buted by U. C. Berkeley. 

It discusses the overall model for interprocess communication and intro¬ 
duces the interprocess communication primitives which have been added to the 
system. The majority of the document considers the use of these primitives in 
developing applications. The reader is expected to be familiar with the C pro¬ 
gramming language as all examples given are w’ritten in C. 










• DEC and VAX are trademarks of Digital Equipment Corporation. 

• ♦ UNIX is a Trademark of Bell Laboratories 
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1. INTRODUCTION 


One of the most important parts of 4.2bsd is the interprocess communication facilities. These 
facilities are the result of more than two years of discussion and research. The facilities provided 
in 4.2bsd incorporate many of the ideas from current research, while trying to maintain the UNIX 
philosophy of simplicity and conciseness. The interprocess communication facilities included in 
4.2bsd are intended to establish a standard for UNIX. From the response to the design, [l], it 
appears many organizations carrying out work with UNIX appear to be adopting the design. 

UNIX has previously been very weak in the area of interprocess communication. Prior to 
the 4.2bsd facilities, the only standard mechanism which allowed two processes to communicate 
has been pipes (the mpx files which were part of Version 7 were experimental). Unfortunately, 
pipes are very restrictive in that the two communicating processes must be related through a 
common ancestor. Further, the semantics of pipes makes them almost impossible to extend to a 

distributed environment. 

Earlier attempts at extending the ipc facilities of UNIX have met with mixed reaction. The 
majority of the problems have been related to the fact these facilities have been tied to the UNIX 
file system; either through naming, or implementation. Consequently, the ipc facilities provided 
in 4.2bsd have been designed as a totally independent subsystem. The 4.2bsd ipc allows processes 
to rendezvous in many ways. Processes may rendezvous through a UNIX file system-like name 
space (a space where all names are path names) as well as through a network name space. In 
fact, new name spaces may be added at a future time with only minor changes visible to users. 

Further, the communication facilities have been extended to included more than the simple byte 
stream provided by a pipe-like entity. These extensions have resulted in a completely new part of 
the system which users will need time to familiarize themselves with. It is likely that 
us made of these facilities they will be refined, only time will tell. 

The remainder of this document is organized in five sections. Section 2 introduces the new 
system calls and the basic model of communication. Section 3 describes some of the supporting 
library routines users may find useful in constructing distributed applications. Section 4 is con¬ 
cerned with the client/server model used in developing applications and includes examples of the 
two major types of servers. Section 5 considers some of the issues related to protection. Section 
6 delves into advanced topics which sophisticated.users are likely to encounter when using the ipc 

facilities. 
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2. BASICS 


The basic building block of communication is the socket . A socket is an endpoint of com¬ 
munication to which a name may or may not be bound . Each socket in use has a type and one or 
more associated processes. Sockets exist within communication domains. A communication 
domain is an abstraction introduced to bundle the underlying assumptions upon which communi¬ 
cation between sockets is predicated. One property emobodied in a communication domain is the 
scheme used in naming sockets. For example, in the UNIX communication domain sockets are 
currently named with UNIX path names; e.g. a socket may be named “/dev/foo”. Sockets nor¬ 
mally exchange data only with sockets in the same domain (it may be possible to cross domain 
boundaries, but only if some translation processs performed). The 4.2bsd ipc supports two 
separate communication domains: the UNIX domain, and the Internet domain for processes which 
communicate using the the DARPA standard communication protocols. The underlying communi¬ 
cation facilities provided by these domains have a significant influence on the internal system 
implementation, as well as the interface to socket facilities provided a user. A notable example of 
the latter is that a socket “operating” in the UNIX domain sees a subset of the possible error con¬ 
ditions which are possible when operating in the Internet domain. 

Typing of sockets is also an important notion. Sockets have been typed according to the 
communication properties visible to a user. Processes are presumed to communicate only between 
sockets of the same type, though in a distributed environment there is nothing that prevents com¬ 
munication between sockets of different types, provided, of course, the underlying communications 

protocols support this. 

Three types of sockets currently are available to a user. A stream socket provides for the 
bidirectional, reliable, sequenced, and unduplicated flow of data without record boundaries. Aside 
from the bidirectionality of data flow, a pair of stream sockets provides an interface nearly identi- £o/\/)<c t'O') 

Onf of boAd 




cal to that of pipes*. 
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A datagram socket supports bidirectional flow of data which is not promised to be sequenced 
or unduplicated. That is, a process receiving messages on a datagram socket may find data repli¬ 
cated and, possibly, arrive out of sequence. An important characteristic of a datagram socket is 
that record boundaries in data are preserved. Datagram sockets closely model the facilities found 
in most contemporary packet switched networks such as the Ethernet. 

A raw socket allows users to access the underlying communication protocols used to support 
the basic socket abstractions. These sockets are normally datagram oriented, though their exact 
characteristics are dependent on the interface provided by the protocol. Raw sockets are not 
intended for the general user; they have been provided mainly for those interested in developing 
new communication protocols, or for gaining access to some of the more esoteric facilities of an 
existing protocol. The use of raw sockets will be considerd in section 6. 

Two potential socket types which have interesting properties are the sequenced packet socket 
and the reliably delivered message socket. A sequenced packet socket is identical to a stream 
socket with the exception that record boundaries are preserved. This interface is very similar to 
that provided by the Xerox NS Sequenced Packet protocol. The reliably delivered message socket 
has the same properties as a datagram socket, but with reliable delivery. While we have loosely 
defined these two socket types, they are currently unimplemented in 4.2bsd. As such, we will 

concern ourselves only with the three socket types for which support exists. 

♦ 

To create a socket the socket system call is used: 


socket(domain, type, protocol); 

This call requests the system create a socket in the specified domain and of the specified type. A 

_ _ , _ _ . _ , ., r 

• In the UNIX domiin, in fact, the semiotics ire identicil md, is one might expect, pipes hive been imple¬ 
mented mternilly is simply i piir of connected sockets 
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particular protocol may also be requested. If the protocol is left unspecified (a value of 0), the 
system will select an appropriate protocol from those protocols which comprise the communica¬ 
tion domain and which may be used to support the requested socket type. The user is returned a 
descriptor (a small integer number) which may be used in later ipc related system calls. The 
domain is specified as one of the manifest constants defined in the file <sys/socket.h>. For the 

UNIX domain the constant is AFJUNIX; for the Internet domain AFJNET. The 6ocket types 
are also defined in this file and one of SOCK_STREAM, SOCKJ)GRAM, or SOCKJRAW. must 
be specified. To create a stream socket in the Internet domain the following call might be used: 

socket(AF JNET, SOCKJ5TREAM, 0); 

This call would result in a stream socket being created with the TCP protocol providing the 
underlying communication support. To create a datagram socket for on-machine use a sample 
call might be: 
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socketfAFJUNIX, SOCKJ)GRAM, 0); 




To obtain a particular protocol one selects the protocol number, as defined within the com¬ 
munication domain. For the Internet domain the available protocols are defined in 
<netinet/in.h> or, better yet, one may use one of the library routines which are discussed in sec¬ 
tion 3. The routine getprotobyname may be used in conjunction with the socket call as follows: 

#include <sys/types.h> 

^include <sys/sockct.h> 

#include <netinet/in.h> 

#include <netdb.b> 
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getprotobynamef’tcp'’); 

socket(AF_INET, SOCKJSTREAM, pp->p_proto); 








There are several reasons a socket call may fail. Aside from the rare occurence of lack of 
memory (ENOBUFS), a socket request may fail due to a request for an unknown protocol 
(EPROTONOSUPPORT), or a request for a type of socket for which there is no supporting pro¬ 
tocol (EPROTOTYPE). 

♦ 

Once a socket has been created, one may perform many operations on it. The most com¬ 
mon of these is to bind a name to the socket. Socket which are to be used as “queueing points 
in a rendezvous with a client process must bind a name to their socket; otherwise the system 
would have no way of identifying the server. The bind call is used for this: 

bind(s, name, namclen); 


11 


p 




The bound name is a variable length byte string which is interpreted by the supporting 
protocol(s). Its interpretation varies from communication domain to communication domain (this 

of the properties which comprise the “domain”). In the UNIX domain names are path 

Internet address and port number. If one 


is one 

names while in the Internet domain names contain 
wanted to bind a UNIX domain socket to the name “/dev/foo”, the following would be used: 


*ii' 


bind(s, ”/dev/foo", sizeof (”/dev/foo”)); 

(Note how the null byte in the name is not counted as part of the name.) In binding 
address things become more complicated. The actual call is simple, 

^include <sys/types.h> 

^include <netinct/in.h> 


Internet 




struct sock add r_jn sin; 


bind(s, &sin, sizeof (sin)); 

but the selection of what to place in the address sin requires some discussion. We will come back 
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to the problem of formulating Internet addresses in section 3 when the library routines used in 
name resolution are discussed. 

With a bound socket it is possible to rendezvous with an unrelated process. This operation 
is usually asymetric with one process a “client" and the other a “server". The client requests ser¬ 
vices from the server by initiating a “connection" to the server’s socket. The server, when willin 
to offer its advertised services, passively “listens" on its socket. On the client side the connect 
call is used to initiate a connection. Using the UNIX domain, this might appear as, 

connects, ’ , se^ver-namc , ’ , , sizeof ("server-name")); 
while in the Internet domain, 

connects, &server, sizeof (server)); 

If the socket is unbound at the time of the connect call, the system will automatically select and 
bind a name to the socket. An error is returned when the connection was unsuccesful. Other¬ 
wise, the socket is associated with the server and data transfer may begin. 

Many errors may be returned when a connection attempt fails, we consider the most com- 


! 






mon: 


j 


EISCONN 


The socket is already connected. The current connection is not affected. 

ETIMEDOUT 

The protocol decided there was no point in retrying the connection attempt anymore. This 
usually occurs because the destination host is down, or because problems in the network 
resulted in transmissions being lost. 

ECONNREFUSED 

The host refused service for some reason. When connecting to a host running 4.2bsd this 

due to the server process not being present at the requested name. 

ENETDOWN or EHOSTDOWN 

These operational errors are returned based on status information delivered to the client 
host by the underlying communication services. 

ENETUNREACH or EHOSTUNREACH 

These operational errors can occur either because the network or host is unknown (no route 
to the network or host is present), or because of status information returned by intermediate 
gateways or switching nodes. Many times the status returned is not sufficient to distinguish 
a network being down from a host being down. In these cases the system is conservative 
and indicates the entire network is unreachable. 


V 
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For the server to receive a client’s connection it must perform two steps after binding its 
socket. The first is to indicate a willingness to listen for incoming connection requests: 

listen(s, 5); 

The second parameter to the listen call specifies the maximum number of outstanding connections 
which may be queued awaiting acceptance by the server process. Should a connection be 
requested while the queue is full, the connection will not be refused, but rather the individual 

packets which comprise the request will be dropped. This gives a harried server time to make 

% 

room in its pending connection queue while the client retrys the connection request. Had the con¬ 
nection been returned with the ECONNREFUSED error, the client would be unable to tell if the 
server was up or not. As it is now it is still possible to get the ETIMEDOUT error back, though 
it is unlikely. The backlog figure supplied with the listen call is quietly chopped by the system to 
allow a maximum of 5 pending connections on any one queue. This avoids the problem of 

processes hogging system resources by setting 
requests. 


infinite backlog, then refusing to ignoring all 




With a socket marked as listening, a server may accept a connection: 
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= sizeof (from); 
accepts, &from, &fromlcn); 

A new descriptor is returned on receipt of a connection (along with a new socket). If the server 
wishes to find out who its client is, it may supply a buffer for the client socket’s name. The 
value-result parameter fromlen is initialized by the server to indicate how much space is associ¬ 
ated with from, then modified on return to reflect the true size of the name. 

Accept is normally a blocking call. That is, the call to accept will not return until a connec¬ 
tion is available or the system call is interrupted by a signal to the process. Further, there is no 
way for a process to indicate it will accept connections from only a specific individual, or indivi¬ 
duals. It is up to the user process to consider who the connection is from and close down the con¬ 
nection if it does not wish to speak to the process. If the server process wants to accept connec¬ 
tions on more than one socket, or not block on the accept call there are alternatives; they will be 

considered in section 6. 

With a connection established, data may begin to flow. To send and receive data there are 
number of possible calls. With the peer entity at each end of a connection anchored, a user can 
send or receive a message without specifying the peer. As one might expect in this case, the nor¬ 
mal read and write system calls are useable, 

write(s, buf, sizeof (buf)); 
read(s, buf, sizeof (buf)); 

In addition to read and write , the new calls send and reev are may be used: 

send(s, buf, sizeof (buf), flags); 
recv(s, buf, sizeof (buf), flags); 

While send and reev are vitually identical to read and write , the extra flags argument is impor¬ 
tant. The flags may be specified as a non-zero value if one or more of the following is required: 


fromlen 

snew = 


\ 




send/receive out of band data 
look at data without reading 
send data without routing packets 


SOFjOOB 

SOF J 3 REVIEW 

SOFJ)ONTROUTE 


Out of band data is a notion specific to stream sockets, and one which we will not immediately 
consider. 


The option to have data sent without routing applied to the outgoing packets is 
tly used only by the routing table management process, and is unlikely to be of interest to 

When SOF PREVIEW is 


curren 

the casual user. The ability to preview data is, however, of interest, 
specified with a reev call, any data present is returned to the user, but treated as still “unread . 
That is, the next read or reev call applied to the socket will return the data previously previewed. 


* 


Once a socket is no longer of interest, it may be discarded by applying a close to the 


descriptor, 


close(s); 

If data is associated with a socket which promises reliable delivery (e.g. a stream socket) when a 
close takes place, the system will continue to attempt to transfer the data. However, after a fairly 
long period of time, if the data is still undelivered, it will be discarded. Should a user find no use 
for any pending data, it may perform a shutdown on the socket prior to closing it. This call is of 

4 

the form: 


shutdowns, &how), 

where how is 0 if the user is no longer interested in reading data, 1 if no more data will be sent, or 
2 if no data is to be sent or received. Applying shutdown to a socket causes any data queued to 

be immediately discarded. 

« 

To this point we have been concerned mostly with sockets which follow a connection 

However, there is also support for connectionless interactions typical of the 


oriented model 
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datagram facilities found in contemporary packet switched networks. A datagram socket provides 
a symetric interface to data exchange. While processes are still likely to be client and server 
there is no requirement for connection establishment. Instead, each message includes the destina 
tion address. 

Datagram sockets are created as before, and each should have a name bound to it in order 
that the receipient of a message may identify the sender. However, to 6end data the $enito primi 
tive must be used 
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sendto(s, buf, buflen, flags, &to, tolen); 

The flags parameter is used as before. The to and tolen values are used to indicate the intended 

receipient of the message. When using an unreliable datagram interface, it is unlikely any errors 

6 

will be reported to the sender. Where information is present locally to recognize a message which 
may never be delivered (for instance when a network is unreachable), the caller will be informed. 

To receive messages on an unconnected datagram socket, the reevfrom primitive is provided: 

c 

revfrom(s, buf, buflen, flags, &from, &fromlen); 

Once again, the fromlen parameter is handled in a value-result fashion, initially containing the 
size of the from buffer. 

In addition to the two calls mentioned above, datagram sockets may also use the connect 
call to associate a socket with a specific address. In this case, any data sent on the socket will 
automatically be addressed to the connected peer, and only data received from that peer will be 
delivered to the user. Other of the less important details of datagram sockets will be described in 
section 6. 
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One last basic facility required to develope applications is the ability to multiplex i/o 

4 

requests among multiple sockets and/or files. This is done using the »elect call: 

a> 

select(nfds, fereadfds, fcwritefds, &execptfds, &timeout); 

arguments three bit masks, one for the set of file descriptors for which the caller 
wishes to be able to read data on, one for those descriptors to which data is to be written, and 
one for which exceptional conditions are pending. Bit masks are created by or-ing bits of the form 
1 < < fd”. That is, a descriptor fd is selected if a 1 is present in the fd *th bit of the mask. The 
parameter nfds specifies the range of file descriptors (i.e. one plus the value of the largest descrip¬ 
tor) specified in a mask. A timeout value may be specified if the selection is not to last more than 
a predetermined period of time. If timeout is set to 0, the selection takes the form of a poll , 
returning immediately. If the last parameter is a null pointer (0), the selection will block 
indefinitely*. Select provides a synchronous multiplexing scheme. Asynchronous notification of 
output completion, input availability, and exceptional conditions is possible through use of the 
SIGIO and SIGURG signals described in section 6. 


Select takes 
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• To bt more specific, a return takes place only wben a descriptor is selectable, or when a signal is received by 
the caller, interrupting the system call. 
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The discussion in section 2 indicated the possible need to locate and construct network 
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i a distributed environment. - To 

run-time library. In this 

e will consider the new routines provided to deal with network addresses. While r the 

these .routines 




this task a number of routines have been added to the standard C 




aid in 
section 


♦ 


a 


* 


t 


i.i 


< 


* 


i 




• k •* 




• • 




7 ft 






C 


I 


have been designed with flexibility in mind. As more communication protocols become available, 
we hope the same user in terface will be maintained jin?accessing ‘ network related address data : 

j to the user. ; Since these values 

mally just supplied the system, users should not need to tie directly aware of the communication 

^ ^ ^ — s 4 a 

protocol and/or naming conventions in use. 
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Locating a service on a remote host requires many levels of mapping before client and 
may communicate. A service is assigned a name which is intended for human consumption; e.g. 

the login server on host monet”. This name, and the name of the peer host, must then be 
translated into network addresses which are hot necessarily suitable for human consumption. 
Finally, the address must then used in locating a physical location and route to the service. The 
specifics of these three mappings is likely to vary between network architectures. For instance, it 
is desirable for a network to not require hosts be be named in such a way that their physical loca¬ 
tion is known by the client host. Instead, underlying services in the network may discover the 

# 

actual location of the host at the time a client ho6t wishes to communicate. This ability to have 

♦ 
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hosts named in a location independent manner may induce overhead in connection establishment, 
a discovery process must take place, but allows hosts to be physically mobile without requiring 
it notify its clientele of its current location. 
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Standard routines are provided for: mapping host names to network addresses, network 
names to network numbers, protocol names to protocol numbers, and service names to port 
numbers and the appropriate protocol to use in communicating with the server process. The file 
<netdb.h> must bejncluded when using any of these routines. 

A host name to address mapping is represented by the hostent structure: 

struct hostent { 

char 
char 
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/* official name of host */ 
/* alias list */ 

/* host address type */ 

/* length of address */ 

/* address */ 


*h_name; 

**h_aliases; 

hjaddrtype; 

hjength; 

*h_addr; 
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int 




int 


char 


; 


The official name of the host and its public aliases are returned, along with a variable length 
address and address type. The routine gethostbyname(SN) takes a host name and returns a hos¬ 
tent structure, while the routine getho$tbyaiir{ 3N) maps host addresses into a hostent structure. 
It is possible for a host to have many addresses, all having the same name. Gethostybyname 
returns the first matching entry in the data base file /etc/hosts; if this is unsuitable, the lower 
level routine getkostent( 3N) may be used. For example,*to obtain a hostent structure for a host 
on a particular network the following routine might be used (only Internet addresses 
sidered, for simplicity): 
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# include <sys/types.h> 
#include <6ys/socket.h> 
^include <n'etinet/in.h> 
#include <netdb.h> ! 
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6truct hostent * 

gethostbynameandnet(name,’ net) 

char *name; 
int net; 
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register struct hostent *hp 
register char **cp; 
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while ((hp = gethostent()) != NULL) { 

if (hp->h_addrtype !== AFJNET) 

continue; 

if (strcmp(name, hp->h_pame)) { 

for (cp = bp- > h_al iases; cp 

if (strcmp(name, *cp) 
goto found; 
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continue; 
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if (in jietof(*(struct in m jAdt *)hp->h jxldr)) 
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break; 
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return (hp); 
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(in_nctoJ[3N) is a standard routine which returns the network portion of an Internet address.) 

As for host names, routines for mapping network names to numbers, and back, are pro¬ 
vided. These routines return a netent structure: 


** 


4 


/* 


♦ Assumption here is that a network number 

♦ fits in 32 bits — probably a poor one. 


4 


/ 
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struct netent { 

char 

char 




/* official name of net */ 
/* alias list ♦/ 

/* net address type */ 


*n_name; 
**n_aliases; 
addrtype; 
net; 




int 


i 


t 


/* network # */ 


int 


I 


}; 


► 




The routines gctnctbynamc{ZN), gctnetbynumber{ 3N), and getneient{ 3N) are the network counter 
parts to the ho6t routines described above. 

- ; • ? . ; 

For protocols the protoent structure defines the protocol-name mapping used with the rou 

• * 

tines getprotobyname( 3N), gctprotobynumbcr(3N) 9 and ^e<pro<aen<(3N): 

struct protoent { 

char 
char 
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/* official protocol name */ 
/* alias list */ 

/* protocol # */ 




♦pjiame; 

**P->liases; 

P-Proto; 


int 


; 




Information regarding services is a bit more complicated. A service is expected to reside at 
a specific “port address” and employ a particular communication protocol. This view is 
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consistent with the Internet world, but inconsistent with other network architectures. Further, a 
service may reside on multiple ports or support multiple protocols. If either of these occurs, the 
higher level library routines will have to be bypassed in favor of homegrown routines similar in 
spirit to the “gethostbynameandnet” routine described above. A service mapping is described by 

the eervent structure 

struct servent { 

char 
char 


i 






! 


/* official service name */ 

I* alias list */ 

/* port # */ 

/* protocol to use */ 


*s_pame; 

♦*s_aliases; 

s_port; 

*s_proto; 


int 


char 


•• 


% 1 


}; 


r 'j 


i 


.* 




The routine jef*erv6yname(3N) maps service names to a 6ervent structure by specifying a service 
name and, optionally, a qualifying protocol. Thus the call 


l 


T 


getservbyname(” telnet”, (char *)0); 

returns the service specification for a telnet server using any protocol, while the call 

a 

getservbyname(” telnet”, ”tcp”); 

returns only that telnet server which uses the TCP protocol. The routines getservbyport( 3N) and 
getservent(3N) are also provided. The getservbyport routine has an interface similar to that pro¬ 
vided by getservbyname; an optional protocol name may be included to qualify lookups. 

With the support routines described above, an application program should rarely have to 
deal directly with addresses. This allows services to be developed as much as possible in a net¬ 
work independent fashion. It is clear, however, that purging all network dependencies is very 
difficult. So long as the user is required to supply network addresses when naming services and 
sockets there will always some network dependency in a program. For example, the normal code 
included in client programs, such as the remote login program, is of the form shown in Figure 1. 

(This example will be considered in more detail in section 4.) 

a 

If one wanted to make the remote login program independent of the Internet protocols and 
addressing scheme we would be forced to add a layer of routines which masked the network 
dependent aspects from the mainstream login code. For the current facilities available in the sys¬ 
tem this does not appear to be worthwhile. Perhaps when the system is adapted to different net¬ 
work architectures the utilities will be reorganized more cleanly. 


\ 




«. 






x 


K 


( 


1 


Aside from the address-related data base routines, there are several other routines available 

These are intended mostly to simplify 


f 


i 

I 


t 


in the run-time library which are of interest to users, 
manipulation of names and addresses. Table 1 summarizes the routines for manipulating variable 

length byte strings and handling byte swapping of network addresses and values. 


The byte swapping routines are provided because the operating system expects addresses to 
be supplied in network order. On a VAX, or machine with similar architecture, this is usually 
reversed. Consequently, programs are sometimes required to byte swap quantities. The library 
routines which return network addresses provide them in network order so that they may simply 

be copied into the structures provided to the system. This implies 
byte swapping problem only when interpreting network addresses. For example, if 

port is to be printed out the following code would be required: 


should encounter the 
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#include <sys/types.h> 

#include <sys/socket.h > 
#include <netinet/in.h> 

#include <stdio.b> 
^include <netdb.h> 
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main(argc, argv) 

char *argv[]; 


* 


{ 


struct sockaddrjn sin; 
struct servent *sp; 
struct hostent *hp; 
int 6; 


getservbyname(”login 


tcp"); 

fprintf(stderr, "rlogin: tcp/login: unknown service\n”); 
exit(l); 








(s 


NULL) { 


I 


d 




hp = gethostbyname(argv[l]); 

NULL) { 

fprintf(stderr, "rlogin: %s: unknown host\n", argv(l|); 

exit(2); 


if ( 


P 




d 




bzero((char *)&sin, sizeof (sin)); 

bcopy(hp->h_addr, (char *)&sin.sin_addr, hp*>h_Jength); 

6in.sin_family = bp->h_>ddrtype; 

sin.sin_port *= 6p->s_port; 

socket(AF_JNET, SOCK_STREAM, 0); 

if (s < 0) { 


N 


perror(”rlogin: socket”); 
exit(3); 


} 


if (connects, (char *)&sin, sizeof (sin)) < 0) { 

perror(” rlogin: connect”); 
exit(5); 






Figure 1. Remote login client code. 


Call 


Synopsis 

compare byte-strings; 0 if same, not 0 otherwise 
copy n bytes from si to s2 
zero-fill n bytes starting at base 

convert 32-bit quantity from VAX to network byte order 
convert 16-bit quantity from VAX to network byte order 
convert 32-bit quantity from network to VAX byte order 
convert 16-bit quantity from network to VAX byte order 


bcmp(sl, 62, n) 
bcopy(sl, s2, n) 
bzero(base, n) 
htonl(val) 
htons(val) 
ntohl(val) 
ntohsfval 


1 


i' 


Table 1. C run-time routines. 
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printf("port number %d\n”, ntobs(sp->6_port)); 


On machines other than the VAX these routines are defined 


% 


null macros. In hindsight it seems 
clear the system should be responsible for any byte swapping problems, with the user dealing only 

with network addresses and numbers in the machine representation. If this is eventually changed, 

programs which use the library routines described here should only require, recompilation, unless 
they perform explicit manipulation of addresses (as above). 
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4. CLIENT/SERVER MODEL 


The most commonly used paradigm in constructing distributed applications is the 

server process. 


client/server model. In this scheme client applications request services from 
This implies an asymetry in establishing communication between the client and server which has 
been examined in section 2. In this section we will look more closely at the interactions between 
client and server, and consider some of the problems in developing client and server applications. 


Client and server require a well known set of conventions before service may be rendered 

• t -1 ^ * p 

(and accepted). This set of conventions comprises a protocol which must be implemented at both 
ends of a connection. Depending on the situation, the protocol may be symetric or asymetric. In 

0 m • 

symetric protocol, either side may play the client or server roles. In an asymetric protocol, one 
side is immutably recognized as the server, with the other the client. An example of a symetric 
protocol is the TELNET protocol used in the Internet for remote terminal emulation. An exam¬ 
ple of an asymetric protocol is the Internet file transfer protocol, FTP. No matter whether the 
specific protocol used in obtaining a service is symetric or asymetric, when accessing a service 
there is a “client process” and a “server process”. We will first consider the properties of server 
processes, then client processes. 

A server process normally listens at a well know address for service requests. Alternative 
schemes which use a service server may be used to eliminate a flock of server processes clogging 
the system while remaining dormant most of the time. The Xerox Courier protocol uses the 
latter scheme. When using Courier, a Courier client process contacts a Courier server at the 
remote host and identifies the service it requires. The Courier server process then creates the 
appropriate server process based on a data base and “splices” the client and server together, void¬ 
ing its part in the transaction. This scheme is attractive in that the Courier server process may 
provide a single contact point for all services, as well as carrying out the initial steps in authenti¬ 
cation (for more about authentication 6ee section 5). However, while this is an attractive possibil¬ 
ity for standardizing access to services, it does introduce a certain amount of overhead due to the 
intermediate process involved. Implementations which provide this type of service within the sys¬ 
tem can minimize the cost of client server rendezvous . The portal notion described in the 
“4.2BSD System Manual" embodies many of the ideas found in Courier, with the rendezvous 

mechanism implemented internal to the system. 

In 4.2bsd most servers are accessed at well known Internet addresses or UNIX domain 
names. When a server is started at boot time it advertises it services by listening at a well know 
location. For example, the remote login server’s main loop is of the form shown in Figure 2. 

The first step taken by the server is look up its service definition: 
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main(argc, argv) 

int argc; 

char **argv; 


{ 


int f; 

struct sockaddrjn from; 
struct servent *sp; 


tcp"); 


getservbyname( w login 


n n 






NULL) { 


if (sp 


fprintf(stderr, "rlogind: tcp/login: unknown serviceO); 
exit(l); 


t 


i 






! 


i 


i 


i 


#ifndef DEBUG 


<<disassociate server from controlling terminal>> 


#endif 


sin.sin_port 


sp->sjport; 


socket(AFJNET, SOCKJSTREAM, 0, 0); 

if (bind(f, (caddrjb)&sin, sizeof (sin)) < 0) { 




i 




} 


listen(f, 10); 

t°r (;;) { 




int g, len = sizeof (from); 


i 


accept(f, &from, &Ien, 0); 
if (g < 0) { 




if (errno != EINTR) 

perrorf”rlogind: accept”); 


continue; 


} 


if (forkQ 


0 


close(f); 
doit(g t &from); 




close(g); 




} 


Figure 2. Remote login server. 

getservbyname(”login”, "tcp”); 




NULL) { 


if (sp 


fprintf(stderr, "rlogind: tcp/login: unknown serviceO); 
exit(l); 


} 


This definition is used in later portions of the code to define the Internet port at which it listens 

for service requests (indicated by a connection). 

Step two is to disassociate the server from the controlling terminal of its invoker. This 
important as the server will likely not want to receive signals delivered to the process group of the 
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controlling terminal. 

#ifndef DEBUG 

if (fork()) 


exit(O); 

for (f — 0; f < 10; f+ +) 

(void) close(f); 

(void) open(*/", 0); 

(void) dup2(0, 1); 

(void) dup2(0, 2); 

{ int t = open(”/dev/tty”, 2); 

if (t >= 0) { 

ioctl(t, TIOCNOTTY, (char *)0); 

(void) close(t); 


} 




#endif 

The steps in this little horror are: 

1) Fork once to allow the calling process to continue its business. 

2) Close file descriptors to insure the next step opens file descriptors of known value (a descrip¬ 
tor is assigned by the system according to the first available; thus, if descriptor 0 is avail¬ 
able, the next open, call will always return descriptor 0) 

3) Open the root file system directory for reading, then duplicate the descriptor to the stan¬ 
dard output and standard error. This is safe (even if the process executes as the super user) 
because the system disallows any attempts to write on directories). Code of this sort has 
been carried over from time in memorium and, while most obscure, is known to work (sigh). 

Clear any process group associated with the server (TIOCNOTTY). If a server allows its 

process group to be “randomly” assigned, it may receive unexpected signals which could 
terminate it. By reseting its controlling terminal, it may manipulate its process group in a 
well defined manner, thereby controlling its own destiny, so to speak. 

The steps described are not carried out when the server is compiled for debugging. This 

allows the user debugging the server to abort servers with an interrupt from the keyboard, and 

% 

also view diagnostic output normally lost because the standard output and error descriptors have 
been closed. A better scheme whereby diagnostics are sent, in messages, to a system logger pro¬ 
cess is planned for the future. 

Once a server has established a pristine evironment, it creates a socket and begins accepting 
service requests. The bind call is required to insure the server listens at its expected location. 
The main body of the loop is fairly simple: 
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for (;;) { 


sizeof (from); 


int g, len 


accept(f, &from, &len, 0); 

if (g < 0) { 




hecw 


f 


o o 


if (errno != EINTR) 

perrorf” rlogind: accept”); 


S/GCHtP 


continue; 




if (fork() 


0 ) 


I 


close(f); 

doit(g, &from); 




close(g); 


} 


An accept call blocks the server until a client requests service. This call could return a failure 
status if the call is interrupted by a signal such as SIGCHLD (to be discussed in section 6). 
Therefore, the return value from accept is checked to insure a connection has really been esta¬ 
blished . With a connection in hand, the server then forks a child process and invokes the main 

i 

body of the remote login protocol processing. Note how the socket used by the parent for queue¬ 
ing connection requests is closed in the child, while the socket created as a result of the accept is 
closed in the parent. The address of the client it also handed the doit routine because it requires 
it in authenticating clients. 

The client side of the remote login service was shown earlier in Figure 1. One can see the 
separate, asymetric roles of the client and server clearly in the code. The server is a passive 
entity, listening for client connections, while the client process is an active entity, initiating a con¬ 
nection when invoked. 

Let’s consider more closely the steps taken by the client remote login process. As in the 
server process, the first step is to locate the service definition for a remote login. 

sp = getservbyname("Iogin”, "tep"); 
if (sp 


f 


NULL) { 


fprintf(stderr, "rlogin: tcp/login: unknown service\n"); 

exit(l); 




Next the destination host is looked up with a gethostbyname call: 


hp = gethostbyname(argv[l]); 

if (hp 


NULL) { 


fprintf(stderr, "rlogin: %s: unknown host\n”, argv(lj); 
exit(2); 


} 


With this accomplished, all that is required is to establish a connection to the server at the 
requested host and start up the remote login protocol. The address buffer is zeroed, then filled in 
with the Internet address of the foreign host and the port number at which the login process 
resides: 


bzero((char *)&sin, sizeof (sin)); 

bcopy(hp->h_addr, (char *)sin.sin_addr, bp->hjength); 
sin.sinjamily = hp->h_addrtype; 
sin .sin.port = sp->s_port; 
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A socket is created, and a connection initiated. 
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The details of the remote login protocol will not be considered here; that part of the initial 
nection protocol which relates to authenticating the client will be considered in section 5. 

While connection-based services are the norm, some services are based on the use of 
datagram sockets. One, in particular, is the “rwho” service which provides users with status 
information for hosts connected to a local area network. This service, while predicated on the 
ability to broadcast information to all hosts connected to a particular network, is of interest 
example usage of datagram sockets. 

An user on any machine running the rwho server may find out the current status of 
machine with the ruptime(l ) program. The output generated is illustrated in Figure 3. 


17 


socket(AFJNET, SOCKJ5TREAM, 0); 

if (s < 0) { 






perror("rIogin: socket”); 
exit(3); 


use 


) 


no 


\ 


if (connects, (char *)&sin, sizeof (sin)) < 0) { 

perror(”rlogin: connect”); 
exit(4); 








con 


an 








arpa up 9:45, 
cad up 2+ 12:04, 

calder up 10:10, 
dali up 2+ 06:28, 

degas up 25+ 09:48 

up 5+ 00:05 
ernie down 0:24 
esvax down 17:04 


5 users, load 1.15, 1.39, 1.31 


8 users, load 4.67, 5.13, 4.59 
0 users, load 0.27, 0.15, 0.14 

9 users, load 1.04, 1.20, 1.65 
0 users, load 1.49, 1.43, 1.41 

users, load 1.51, 1.54, 1.56 


rvi 


I 


ear 


•f f./c 

mi 9. / 


L 7J 




ingres down 0:26 

kim 




8 users, load 2.03, 2.46, 3.11 
0 users, load 0.03, 0.03, 0.05 
2 users, load 0.35, 0.37, 0.50 


up 3+ 09:16, 
matisse up 3+ 06:18, 
medea up 3+ 09:39, 
merlin down 19-f 15:37 


miro up 1+ 07:20 
monet up 1+ 00:43 

down 16: 

statvax up 2 + 15:57 
uebvax up 9:34, 


7 users, load 4.59, 3.28, 2.12 
2 users, load 0.22, 0. 


0.07 


•JBE 






3 users, load 1.52, 1.81, 1.86 

2 users, load 6.08, 5.16, 3.28 

Figure 3. ruptime output. 

♦ 

Status information for each host is periodically broadcast by rwho server processes on each 
machine. The same server process also receives the status information and uses it to update a 
database. This database is then interpreted to generate the status information for each host. 
Servers operate autonomously, coupled only by the local network and its broadcast capabilities. 

t 

The rwho server, in a simplified form, is pictured in Figure 4. There are two separate tasks 
performed by the server. The first task is to act as a receiver of status information broadcast by 
other hosts on the network. This job is carried out in the main loop of the program. Packets 
received at the rwho port are interrogated to insure they’ve been sent by another rwho server pro- 

f • 

cess, then are time stamped with their arrival time and used to update a file indicating the status 
of the host. When a host has not been heard from for an exended period of time, the database 
interpretation routines assume the host is down and indicate such on the status reports. This 

algorithm is prone to error, but serves our current needs (one should not, however, depend on this 
information for a true host status). 
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main() 




sp = getservbyname( w who", *udp w ); 
net *=s getnetbyname( w localnet"); 

sin.sin_addr »inetjnakeaddr(INADDR_ANY, net); 

sin.sin_port = sp->s_port; 


socket(AFJNET, SOCKJ5GRAM, 0); 




bind(s, &sin, sizeof (sin)); 


sigset(SIGALRM, onalrm); 
ona!rm(); 


tor (;;) { 


struct whod wd; 
int cc, whod, len 


sizeof (from); 


recvfrom(s, (char *)&wd, sizeof (struct whod), 0, &from, &len); 
if (cc <= 0) { 




if (cc < 0 && errno !== EINTR) 

perror( w rwhod: recv"); 


continue; 




if (from.sin_port != sp->s.j>ort) { 

fprintf(stderr, "rwhod: %d: bad from port\n 

ntohs(from.sin_port)); 




continue; 




if (!verify(wd.wd_hostname)) { 


fprintf(stderr, "rwhod: malformed host name from %x\n 

from.sin_addr); 




continue; 


} 


(void) sprintf(path, "%s/whod.%s’ , > RWHODIR, wd.wdjiostname); 
whod 


creat(path, 0666); 


(void) time(&wd.wd_jecvtime); 
(void) write(whod, (char *)&wd, cc); 
(void) close(whod); 




} 


Figure 4. rwho server. 

The second task performed by the server is to supply information regarding the status of its 
host. This involves periodically acquiring system status information, packaging it up in a packet 
and broadcasting it on the local network for other rwho servers to hear. The supply function is 

triggered by a timer and runs off a signal. Locating the system status information is somewhat 

* * 

involved, but uninteresting. Deciding where to transmit the resultant packet does, however, indi¬ 
cates some problems with the current protocol. 

Status information is broadcast on the local network. For networks which do not sup¬ 
port the notion of broadcast another scheme must be used to simulate or replace broadcasting. 
One possibility is to enumerate the known neighbors (based on the status received). This, 
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unfortunately, requires some “bootstrapping” information, as a server started up on a “quiet 
network will have no known neighbors and thus never receive, or send, any status information. 
This is the identical problem faced by the routing table management process in propagating rout¬ 
ing status information. The standard solution, unsatisfactory as it may be, is to inform one or 
more servers of known neighbors and request that they always communicate with these neighbors. 

9 

If each server has at least one neighbor supplied it, status information may then propagate 
through a neighbor to hosts which are not (possibly) directly neighbors. If the server is able to 
support networks which provide a broadcast capability, as well as those which do not, then net¬ 
works with an arbitary topology may share status information*. 

The second problem with the current scheme is that the rwho process services only a single 
local network, and this network is found by reading a file. It is important that software operating 
in a distributed environment not have any site-dependent information compiled into it. This 
would require a separate copy of the server at each host and make maintenance a severe 
headache. 4.2bsd attempts to isolate host-specific information from applications by providing sys¬ 
tem calls which return the necessary information!. Unfortunately, no straightforward mechanism 
currently exists for finding the collection of networks to which a host is directly connected. Thus 
the rwho server performs a lookup in a file to find its local network. A better, though still unsa¬ 
tisfactory, scheme used by the routing process is to interrogate the system data structures to 
locate those directly connected networks. A mechanism to acquire this information from the sys¬ 
tem is likely to be added to the system. 


It 




r 


1 


• One must, however, be concerned about “loop*”. That is, if a host is connected to multiple networks, it will 
receive status information from itself. It most be careful not to consider itself as a neighbor and attempt to ac¬ 
tively exchange information with itself for this would result in an endless exchange of messages, 
t An example of such a system call is the call which returns the host’s “official 
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Authentication and Security 


6. AUTHENTICATION AND SECURITY 


In a distributed environment processes often must know the identity of a client. This infor¬ 
mation may be used for access control, for accounting, etc. The problem of authenticating client 
processes in such 

developed. The first is based on the involvement of a trusted partner. One process is informed of 
the identity of its client by some trusted party such as the operating system. In shifting the prob¬ 
lem from client/server to a single, but possibly distributed, entity one arrives at a model based on 
capability, or rights, being passed internal to a system. 


environment. has been researched extensively with two major schemes 




t 


The alternative to internalized capabilities is to require each process maintain its 


rights 

external to any partner such as the operating system. Authentication is then performed by a 

server verifying the acceptability of capabilities held by a client. Schemes of this sort require that 

the information used for authentication purposes be encrypted to prevent unwarranted theft of 
service capabilities. 




Schemes based on internalized capabilities inevitably rely on externalized capabilities at 
some level for authenticating each distributed part of the trusted entity. No matter whether 
internalized or externalized capabilities are used for authenticating clients, once a connection has 
been established messages exhanged between trusted parties must be safeguarded against 

ranted eavesdropping and spoofing. All this requires that for a secure environment encryption 
techniques must be used. Unfortunately, the available hardware for performing encryption is far 
too slow to provide acceptable performance in constructing a secure distributed environment. 

Given the lack of hardware support for providing secure externalized (or internalized) capa¬ 
bilities in a distributed environment, 4.2bsd makes no attempt at providing such a facility. 
Instead, the services developed allow users to construct applications which use a restricted form of 
internalized capabilities based on the belief that a network is physically secure. That is, most of 
the services which are part of 4.2bsd assume that clients access a network through another, 
equally trusted, system. This assumption is obviously faulty on a network including personal 
workstations, for network integrity is easily compromised by a user at a workstation subverting 
the local authentication mechanisms in order to masquerade as another identity. 

This section will consider those authentication techniques used by the 4.2bsd standard 

vices, and consider their limitations. One of the goals of 4.2bsd is to provide a basis for continued 

research in security in a distributed environment. As practical solutions are developed they will 
be incorporated in the system*. 


r 


unwar 


% 


ser 


There are two schemes used for authenticating clients. The first requires the client to 
tially execute a login before gaining access to a service. That is, the client must provide 
name and password (encypted) on the remote host. This information is then verified against the 

services 


essen 


a user 


information stored in the public pa$$u;(/(5) file. If the validation succeeds, the requested 
are made available and remote execution takes place in the protection domain of the user’s 
account (i.e. the server performs setuid and setgid calls to establish the server’s protection to be 
that of the supplied user). This form of authentication is as safe as the password supplied, but is 
slow due to the encryption required to authenticate passwords. The rexec{ 3) library routine pro¬ 
vides a standard facility for executing commands on remote machines and using passwords 
authentication mechanism. 


as an 


The second form of authentication is based on the assumption the network is physically 
secure. In this scheme all port numbers in the range 0-1023 are reserved for the super user. That 
is, only a super user may bind a port number to a socket in this range. By depending on this 
fact, and the assumption the cable is free of spoofing, the server side then verifies it is negotiating 
with a trusted client process by checking the port number of the source address of a connection 

* For now, usei* concerned about the safety of information accessable in a distributed environment should keep 
their files encrypted, and/or not run those servers which authentication mechanism described in thb section. 
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(or received packet). If the source address indicates it includes a port number in the priviledged 
range, any information passed to the server is believed to be correct. The remote login and 
remote shell servers utilize code of the form shown in Figure 5 for this purpose. The client's 
address is first checked to verify it is associated with a known host (i.e. that it is present in the 

4 

host name data base). Then, the source port and address format are checked for correctness. 
Should any of the checks fail, the server refuses service. 


r 


doit(f, fromp) 


int f; 

struct sockaddrjn *fromp; 




register struct hostent *hp; 

• • • 

fromp->sin_port = htons((u_short)fromp->sin_port); 
hp = gethostbyaddr(&fromp->sin_addr, sizeof (struct in_addr), 

fromp- > sin_f amily); 

fatal(f, "Host name for your address unknown"); 
if (fromp->sin_/amily != AFJNET || 

fromp->sin_port >= IPPORT_RESERVED) 

fatal(f, "Permission denied”); 


if (hp == 0) 




Figure 5. Authentication using reserved ports. 

Because the basic authentication mechanism is so lax 9 secondary security facilities were 
added to restrict the set of hosts and users that are allowed to use the *'authentication based on 
reserved port numbers” scheme. The file /etc/hosts.equiv on a client host defines the collection of 
hosts which are assumed to be under a single administrative control. These hosts must share 
common user names to allow these names to be passed through, and mapped into user identifiers 
without confusion. With the exception of the super user (and the .rhosts file to be described 
next), when authentication is performed at the server, if the client host is found in the 
equivalence file, it is allowed to use reserved port numbers as an authentication mechanism. Oth¬ 
erwise, permission is denied (remote shell execution), or a password is required from the client 
(remote login). If the intended identity is the super user, or the user has a .rhosts file in their 
home directory, the information present in the .rhosts file is used to specify 
That is, the .rhosts file contains a list of host-user pairs. If the client host-user identification 
matches one of the specifications present in .rhosts, authentication continues; otherwise permission 
is denied. This allows user on equivalenced machines to restrict access to their accounts individu¬ 
ally . 
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. ADVANCED TOPICS 


3 


A number of facilities have yet to be discussed. For most users of the ipc those mechanisms 
already described will suffice in constructing distributed applications. However, others will find 
need to utilize some of the less common features which we will consider in this section. 

The stream socket abstraction includes the notion of “out of band” data. Out of band data 
is an entirely separate transmission channel associated with each pair of connected stream sockets. 
Out of band data is delivered to the user independently of normal data along with the SIGURG 
signal. In addition to the information passed, a mark is placed in the data stream to indicate the 
point at which the out of band data was sent. The remote login and remote shell applications use 
this facility to propagate signals from between client and server processes. When the signal is 
expected to flush any pending output from the remote process(es), all data up to the mark in the 
data stream is discarded. The stream abstraction defines that the out of band data facilities must 
allow at least one, reliably delivered, out of band message be pending at any time. Further, at 
least one byte of data must be able to be sent with each out of band notification. To send an out 
of band message the SOFjOOB flag is supplied to a send or scndto calls, while to receive out of 
band data SOFjOOB should be indicated when performing a recvfrom or recv call. To find out if 
read pointer is currently pointing at the mark in the data stream, the SIOCATMARK ioctl is pro¬ 
vided: 


i 


t 




I 


* 


ioctl(s, SIOCATMARK, &yes); 

If |its is a 1 on return, the next read will return data after the mark. Otherwise (assuming out of 
band data has arrive), the next read will provide data sent by the client prior to transmission of 
the out of band signal. The routine used in the remote login process to flush output on receipt of 
an interrupt or quit signal is shown in Figure 5. 

oob() 


r 

I 


{ 


i 


int out 

char wastc[BUFSIZ], mark; 


i+i; 


% 


i 


signal(SIGURG, oob); 

ioctl(l, TIOCFLUSH, (char *)&out); 

for (;;) { 


I 


if (ioctl(rem, SIOCATMARK, &mark) < 0) { 

perror(”ioctr); 

break; 


g 


y 




if (mark) 


break; 

(void) read(rem, waste, sizeof (waste)); 


t 




recv(rem, &mark, 1, SOFjOOB); 




Figure 5. Flushing input on receipt of out of band data. 

9 

Due to the existence of the SIGURG signal each socket has an associated process group (just 
as is done for terminals). This process group is initialized to the process group of its creator, but 
may be redefined at a later time with the SIOCSPGRP ioctl: 

ioctl(s, SIOCSPGRP, &pgrp); 

A similar ioctl, SIOCGPGRP, is available for determining the current process group of a socket. 
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Many programs will not function properly without a terminal for standard input and out¬ 
put. Since a socket is not a terminal, it ; 
the network do 


often necessary to have a process communicating over 

through a pseudo terminal. A pseudo terminal is actually a pair of devices, 

active agent in communication between 


« 


.j] 


master and slave, which allow a process to serve 
processes and users. Data written on the 6lave side of a pseudo terminal is supplied as input to a 
process reading from the master side. Data written on the master side is given the slave as input. 
In this way, the process manipulating the master side of the pseudo terminal has control over the 
information read and written on the slave side. The remote login server uses pseudo terminals for 
remote login sessions. A user logging in to a machine across the network is provided a shell with 
slave pseudo terminal as standard input, output, and error. The server process then handles the 
communication between the programs invoked by the remote shell and the user’s local client pro- 

. When a user sends an interrupt or quit signal to a process exexcuting on a remote machine, 




j 




cess 

the client login program traps the signal, sends an out of band message to the server process who 
then uses the signal number, sent as the data value in the out of band message, to perform a 

Jh7/(2) on the appropriate process(es). 
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